bench: evaluate on the held-out split, not the training data by chiruu12 · Pull Request #160 · UnplugAI/Unplug

chiruu12 · 2026-08-25T16:30:33Z

benchmarks/download.py pulled neuralchemy/Prompt-injection-dataset with
split="train" and no config. That is the split unplug-tiny was fine-tuned on, so
the headline detection numbers in BENCHMARKS.md and the README were scoring the
model against its own training data.

The row counts line up exactly. HF reports core as train 4391 / validation 941 /
test 942, and the page reported 4,391 samples. The model card gates on
neuralchemy_test_doc_recall, which is the 942-row test split.

Re-measured on core/test, both modes, --isolated:

Mode	F1	Recall	FPR	Precision
regex-only	0.519	0.351	0.0026	0.995
regex + ML	0.974	0.958	0.0128	0.991

Against the published 0.987 / 0.981, contamination was worth about 2.3 points of
recall. Regex-only moved as well, 0.405 down to 0.351, and regex has no training data,
so some of the gap is just that the splits are not equally hard. Both docs say so.

The FPR change matters more than the recall change. The README claimed the
false-positive rate "stays under 1%". It is 5 false positives in 390 benign holdout rows, and the model card's
own public-benchmark axes are far worse: 18.8% on Deepset (662), 54.2% on WildGuard benign (971),
34.1% on a combined 3,227-prompt validation set. Those numbers were only on Hugging
Face. They are in BENCHMARKS.md now, under a section that says to tune the threshold
before putting this in front of users.

Changes:

download.py pins the config to core and defaults to the test split, with a
--neuralchemy-split flag for anyone who deliberately wants train.
BENCHMARKS.md gets the holdout numbers, a note on what changed, and the public
benchmark table.
EVAL_PHASE_C.md keeps its numbers as a record of that run and gets a superseded
note, since rewriting a dated report would be worse.
README benchmark paragraph matches.

microsoft/llmail-inject is unchanged. I have not checked it for overlap with
fine-tuning data, so BENCHMARKS.md now says to read that row as an upper bound
rather than claiming it is clean.

Verified: ruff check and ruff format clean on the changed file, mkdocs build --strict exit 0, 1250 tests pass. The two tests/unit/ml/test_model_store.py
failures reproduce on dev and are a local cached-model artifact, not from this
branch.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

greptile-apps · 2026-08-25T16:33:11Z

Greptile Summary

The PR moves the Neuralchemy benchmark from contaminated training data to the held-out core/test split and updates the published results accordingly.

Defaults the benchmark downloader to the core/test split while retaining explicit split selection.
Replaces superseded metrics in both package-facing READMEs and documents the corrected methodology.
Preserves the earlier Phase C report as historical data with a prominent superseded notice.
Adds broader false-positive benchmark results and deployment-threshold guidance.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
README.md	Replaces contaminated headline metrics and the misstated benign denominator with held-out results and explicit counts.
sdk/README.md	Fully replaces the previously reported 4,391-row training results with consistent `core/test` metrics and caveats.
sdk/benchmarks/download.py	Pins Neuralchemy to the `core` configuration, defaults to `test`, and exposes a validated split-selection option.
sdk/docs/BENCHMARKS.md	Documents the held-out evaluation, corrected metrics, historical discrepancy, and broader false-positive behavior.
sdk/docs/EVAL_PHASE_C.md	Retains the dated training-split results while clearly marking them superseded and linking to current metrics.

_{Reviews (2): Last reviewed commit: "retract contaminated numbers from the pa..." | Re-trigger Greptile}

greptile-apps · 2026-08-25T16:33:14Z



-On the neuralchemy prompt-injection set, regex-only detection reaches **F1 0.58 / recall 0.41** — a fast first line, not sufficient alone. Adding the ML span model (`Guard(model="tiny")`) takes that to **F1 0.99 / recall 0.98**, and lifts recall on *indirect* injection from **0.05 → 0.91**. False-positive rate stays under 1% on the injection set (2.1% on a separate hard-benign corpus). Full tables, methodology, and honest caveats: [`sdk/docs/BENCHMARKS.md`](sdk/docs/BENCHMARKS.md). Per-axis model metrics (including failure modes) are on the [model card](https://huggingface.co/Unplug-AI/unplug-tiny-v1).
+On the held-out `core/test` split of the neuralchemy prompt-injection set (942 rows), regex-only detection reaches **F1 0.52 / recall 0.35**, a fast first line and not sufficient alone. Adding the ML span model (`Guard(model="tiny")`) takes that to **F1 0.97 / recall 0.96**, and lifts recall on *indirect* injection from **0.05 to 0.91**. False positives on that split run at 5 of 390 benign rows, and 2.1% on a separate 135-prompt hard-benign corpus. On broader public benign sets the model over-flags badly, up to 34% of a combined 3,227-prompt validation set, so tune the threshold for your own traffic rather than trusting the defaults. Full tables, methodology, and the axes where it fails: [`sdk/docs/BENCHMARKS.md`](sdk/docs/BENCHMARKS.md). Per-axis model metrics are on the [model card](https://huggingface.co/Unplug-AI/unplug-tiny-v1).


Superseded metrics remain published

When users read the package-facing sdk/README.md, it still presents the 4,391-row training-split results as current, causing the contaminated 0.99 F1 and 0.98 recall figures that this PR retracts to remain publicly advertised.

Knowledge Base Used: Audit and security evaluation

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

github-actions · 2026-08-25T16:33:37Z

SDK Coverage •

File	Stmts	Miss	Cover	Missing
TOTAL	7396	895	88%

report-only-changed-files is enabled. No files were changed during this commit :)

Tests	Skipped	Failures	Errors	Time
1282	25 💤	0 ❌	0 🔥	39.216s ⏱️

benchmark on the held-out split, not train data

5b51719

Copilot AI lite review requested due to automatic review settings August 25, 2026 16:30

Copilot AI reviewed Aug 25, 2026

github-actions Bot added the documentation Improvements or additions to documentation label Aug 25, 2026

greptile-apps Bot reviewed Aug 25, 2026

View reviewed changes

retract contaminated numbers from the package readme

c8ddeb8

chiruu12 merged commit bba131a into dev Aug 25, 2026
9 checks passed

Navigation Menu

Uh oh!

bench: evaluate on the held-out split, not the training data - #160

bench: evaluate on the held-out split, not the training data#160
chiruu12 merged 2 commits into
devfrom
fix/benchmark-holdout-split

chiruu12 commented Aug 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

greptile-apps Bot commented Aug 25, 2026 •

edited

Loading

Important Files Changed

Uh oh!

greptile-apps Bot Aug 25, 2026

Uh oh!

Uh oh!

github-actions Bot commented Aug 25, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants



		On the neuralchemy prompt-injection set, regex-only detection reaches F1 0.58 / recall 0.41 — a fast first line, not sufficient alone. Adding the ML span model (`Guard(model="tiny")`) takes that to F1 0.99 / recall 0.98, and lifts recall on indirect injection from 0.05 → 0.91. False-positive rate stays under 1% on the injection set (2.1% on a separate hard-benign corpus). Full tables, methodology, and honest caveats: [`sdk/docs/BENCHMARKS.md`](sdk/docs/BENCHMARKS.md). Per-axis model metrics (including failure modes) are on the [model card](https://huggingface.co/Unplug-AI/unplug-tiny-v1).
		On the held-out `core/test` split of the neuralchemy prompt-injection set (942 rows), regex-only detection reaches F1 0.52 / recall 0.35, a fast first line and not sufficient alone. Adding the ML span model (`Guard(model="tiny")`) takes that to F1 0.97 / recall 0.96, and lifts recall on indirect injection from 0.05 to 0.91. False positives on that split run at 5 of 390 benign rows, and 2.1% on a separate 135-prompt hard-benign corpus. On broader public benign sets the model over-flags badly, up to 34% of a combined 3,227-prompt validation set, so tune the threshold for your own traffic rather than trusting the defaults. Full tables, methodology, and the axes where it fails: [`sdk/docs/BENCHMARKS.md`](sdk/docs/BENCHMARKS.md). Per-axis model metrics are on the [model card](https://huggingface.co/Unplug-AI/unplug-tiny-v1).

Uh oh!

Conversation

chiruu12 commented Aug 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

greptile-apps Bot Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

greptile-apps Bot commented Aug 25, 2026 •

edited

Loading

github-actions Bot commented Aug 25, 2026 •

edited

Loading